Administration of Lookup Lists: lutserver & emulutsrebuild
The lutserver background service ensures that Lookup List entries are always in sync with the records in the EMu system. This service removed the need in earlier versions of EMu to rebuild the Lookup Lists on a nightly basis.
The Lookup List rebuild program is still provided but now applies changes to the eluts table rather than rebuilding the table completely. The Lookup List service means that rebuilds are only required if the Lookup List table becomes corrupted, or if users accidentally delete records that are still in use.
Although in theory Lookup List tables no longer need to be rebuilt, there may be occasions when they need to be updated manually. It would be useful for example to be able to regenerate a missing entry if someone deleted a record accidentally.
The emulutsrebuild server-side program visits each EMu record and checks that the values in the records are in the Lookup List table. If a value is not present, a new entry is created. Any entries that are no longer used, are deleted unless they are Persistent.
Here you will find details about the lutserver background service and how to rebuild a Lookup List if required:
The server that handles updating of the eluts table in the Lookup List service is called lutserver. lutserver runs on the EMu server machine, waiting for requests to update Lookup List entries. The requests are generated by an audit trail plugin that picks up all deleted and modified records and determines what Lookup Lists need to be updated.
The process may be shown diagrammatically as:
Audit Server (utils/auditserver)
¯
Lookup List Audit Plugin (etc/audit/luts.pl)
¯
Lookup List Service Data (loads/luts/data/*)
¯
Lookup List Server (utils/lutserver)
¯
Lookup List Table (eluts)
In this example a user changes the city value in a record from Bendigo
to Sandhurst and saves the record. The following steps occur:
- When the user saves the record, the EMu server generates an XML description of the changes made to the record. The Audit Server (auditserver) loads these changes and passes them on to all registered plugins. The Lookup List Services registers the Lookup List Audit Plugin when the Audit Server is started. For our example record, the changes will show that
Bendigo
was changed to Sandhurst in the City column. - The Lookup List Audit Plugin looks at the changes supplied by the Audit Server. For each column changed it checks to see if a Lookup List is associated with the column. For our example, it will determine that the City column is associated with the Location Lookup List. It will then look at the two values and their associated operations. The value
Bendigo
has a delete operation (since it is being removed) and the value Sandhurst has an insert operation (as it is the new value). Since the Windows client was used to save the record, the Audit Plugin will ignore the insert operation as the user will have been asked to add the new entry to the Location Lookup List if it did not exist already. The delete operation is passed to the Lookup List Server for processing. - The Lookup List Audit Plugin writes a file containing the:
- Operation performed (delete)
- Value deleted (Bendigo)
- Lookup List Name (Location)
- Column changed (City)
The file is located in the loads/luts/data directory. The format of the file name is date.time.table.irn, where:
- date is an eight digit date in yyyymmdd format.
- time is a six digit time in hhmmss format.
- Table is the database table in which the record was modified.
- irn is the key number of the record modified.
An example file name is: 20121026.133308.ecatalogue.375.
- The Lookup List Server fetches all files in the loads/luts/data directory and sorts them into date/time order. It is important that the files are processed in the same order as they were created, otherwise synchronization issues may arise. The Lookup List Server reads the contents of each file, extracting the information within. Then for each Lookup List in the file it determines the operation to apply (insert or delete):
- For an insert operation it checks to see if the value is already in the Lookup List; if not, it inserts a new value into the eluts table.
- For a delete operation it checks whether the value is used anywhere in the EMu system for the given Lookup List name. If the value is not used, the record is deleted from the eluts table (provided it is not Persistent).
Once the file has been processed, it is removed. Once all the files have been processed, the Lookup List Server waits for new files to process.
The reason for so many steps is that audit plugins must be fast so that the Audit Server can process audit records quickly. To ensure the audit plugin is fast, the main Lookup List processing is removed from the audit plugin and moved to the Lookup List Server. The split ensures that the Audit Server keeps up with audit changes, even when the Lookup List Server may lag.
The Lookup List Server handles all deletion operations, that is it checks whether a Lookup List entry is still in use and if not, deletes it. It also handles insertions (that is new values) from all sources except the Windows client.
The Lookup List Server complies with the Lookup Registry entry and Lookup Exact Registry entry:
- If a column has Lookup Exact set to
true
, then all comparisons with existing entries in the Lookup List table are performed as exact matches, i.e. the character case and punctuation must match exactly. - If Lookup Exact is not enabled, all punctuation and character case is ignored for value comparisons.
The Lookup Registry entry is used to control the flags set when adding new values to a Lookup List. The table below lists each setting and examines how the Lookup List Server implements the required functionality. The Lookup Registry entry settings are generally set on a per column basis.
Setting |
Description |
---|---|
Skip
|
The entry will not be checked against the Lookup List and will not be added to the Lookup List. If the column with For example: we have a Lookup List called Location with a hierarchy of Country>State>City If a new entry of Australia>Victoria>Melbourne is added, then three records would typically be added to the Location Lookup List: Australia Australia> Australia> If Australia Australia> |
|
Values entered into the field must exist in the Lookup List: a new entry will not be added to the Lookup List. |
ReadWrite
|
The |
|
The |
|
If the entry does not exist, a new entry is created with Hidden enabled. If an entry already exists and the Used flag is disabled, it is enabled. |
Finally, the Lookup List Server will not delete any entry from the Lookup List table that has Persistent enabled.
The Lookup List Server, lutserver, provides the functionality required to keep the Lookup List table in sync with values used within the EMu System. The addition of the server removes the need to rebuild the Lookup List tables on a nightly (or otherwise) basis.
The Luts directory is a directory on the EMu server. It contains text files with default values for specific Lookup Lists. During the Rebuild Lookup Lists process, all values in these text files are added to their respective Lookup Lists.
The text files in the Luts directory can be used to import multiple values such as locations and / or post codes into a Lookup List. These values will be added to the Lookup List irrespective of data in that field in the table.
Prior to EMu 4.1, the emulutsrebuild server-side program rebuilt Lookup List tables on a nightly basis, keeping values in Lookup Lists in sync with values keyed that day into fields or added to a Lookup List record in the Lookup Lists module.
The introduction of the Lookup List background service (lutserver) with EMu 4.1 means that Lookup List entries are now always in sync with the records in the EMu system, removing the need to rebuild the Lookup Lists on a nightly basis.
Note: The Lookup List background service (lutserver) means that rebuilds are only required if the Lookup List table becomes corrupted, or if users accidentally delete records that are still in use.
Although in theory Lookup List tables no longer need to be rebuilt, there may be occasions when they need to be updated manually. It would be useful for example to be able to regenerate a missing entry if someone deleted a record accidentally.
The emulutsrebuild server-side program has been rewritten to work with the Lookup List Server framework. The program no longer generates a data file with all Lookup List entries in it that is then loaded into an empty Lookup List table. Instead, the program now visits each EMu record and checks that the values in the records are in the Lookup List table. If a value is not present, a new entry is created. Any entries that are no longer used, are deleted unless they are Persistent.
The rebuild process occurs in two stages. The first stage involves generating a list of all changes that need to be applied to the Lookup List table. This stage is known as the data generation phase. There are three types of changes possible:
- insert - new records to be added to the Lookup List table.
- update - existing records that require their flags (Hidden, Used, Persistent) to be adjusted.
- delete - existing records that are no longer required.
The data for each of the above changes is placed in the Luts directory in files named data.insert, data.update and data.delete respectively.
Once the data generation phase is complete, the loading phase is executed. In this phase the files generated are loaded into the Lookup List table. Once the load is finished, the Lookup List table rebuild is complete.
The usage message for emulutsrebuild is:
Usage: emulutsrebuild [-dlmqtv] [lookup list ...]
|
only produce lookup table data loading files |
|
load lookup changes quickly (takes eluts offline) |
|
only load lookup table data file |
|
include lookup list text files in rebuild |
|
verbose mode, print debugging information |
The following table describes each option:
Option | Description | |||||
---|---|---|---|---|---|---|
|
Run the data generation phase only: the data files to be loaded are generated but not applied. |
|||||
|
Skip the data generation phase and run the loading phase only. The data files to be loaded are assumed to exist. |
|||||
|
Determine whether the Lookup List table should be taken offline while the loading phase is underway. If the |
|||||
|
Useful for tracing what the data generation phase is producing. Each entry to be inserted, updated or deleted is printed to the screen (as well as added to the appropriate data files). |
|||||
-t
|
Informs emulutsrebuild to include text files found in the # # Lines beginning with a hash character are comments # [-]lookup list name|value 1[|value 2|...][=sortorder] where: |
|||||
|
Where a hierarchy entry is specified, it is not necessary to add the upper levels of the hierarchy as these are added automatically. For example, if a text file contained:
Location|Australia|Victoria|Bendigo
then three entries are generated. The entries are the same as specifying:
Location|Australia
Location|Australia|Victoria
Location|Australia|Victoria|Bendigo
The use of text files provides a convenient mechanism for pre-loading Lookup Lists. As stated above, each entry loaded will have Persistent enabled, meaning the entry will persist even if it is not used. The only way to delete such an entry is to locate the record in the Lookup List module and delete it manually.
A System Administrator may configure a table to be skipped when Lookup Lists are being rebuilt. If a line of the form:
RELUTS=no
is found in a file called emuoptions in the database directory, emulutsrebuild will not check data in that table.
The arguments to emulutsrebuild is a list of Lookup List names to rebuild. If a list is not supplied, then all Lookup Lists are checked. The following examples cover some of the uses of emulutsrebuild:
The command used to rebuild all Lookup Lists is:
emulutsrebuild -t
It should not be necessary to run this command. The Lookup List Server maintains Lookup List synchronization. The command may be used to ensure the Lookup List table is synchronized correctly.
The command used to check whether the Lookup List table is synchronized is:
emulutsrebuild -d -t -v
This command runs the data generation phase only. The Lookup List table is not modified. The verbose option prints out any inconsistencies found.
The command used to rebuild the Location and Admin Names Lookup Lists is:
emulutsrebuild Location 'Admin Names'
This command rebuilds the Location and Admin Names Lookup Lists. As the -t
option is not specified, any entries in text files will not be loaded.
Note: Note the use of single quotes to enclose the names of Lookup Lists that contain spaces.